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Introduction 


Clinical  Looking  Glass  (CLG)  is  a  medical  surveillance  and  quality  improvement  software  system  that  provides 
actionable  health  care  metrics  on  longitudinal  patterns  of  care  generated  from  a  data  repository  of  clinical  and 
administrative  information.  It  is  in  production  use  at  Montefiore  Medical  Center  (MMC)  in  New  York  City.  CLG 
fosters  a  cultural  environment  that  rewards  initiative  while  still  protecting  patient  privacy  and  standardization  of 
analysis. 

This  project  creates  a  proof-of-concept  implementation  of  Clinical  Looking  Glass  housed  in  a  secure  military  data 
center  and  mapped  to  Military  Health  Service  (MHS)  Protected  Health  Information  (PHI)  data.  This  proof  of  concept 
will  be  evaluated  by  clinical  users  at  Walter  Reed  National  Military  Medical  Center  and  National  Naval  Medical 
Center  Bethesda.  In  support  of  this  objective,  the  project  includes  development  of  training  materials  and  formal  end 
user  training.  Also  included  are  environmental  and  application  security  reviews  and  mitigation  of  critical  security 
risks,  and  support  of  the  development  of  additional  application  features  that  will  benefit  the  MHS  user  community. 


Body 


Overview 

Personnel 

The  project  team  experienced  some  personnel  churn  during  the  reporting  period.  Effort  was  redistributed  among 
remaining  team  members  after  departures,  and  new  team  members  were  brought  on  board  as  efficiently  as  possible. 
All  persons  with  need  to  access  the  proof-of-concept  implementation  housed  at  SPA  WAR  have  been  cleared  for  a 
position  of  public  trust,  including  new  team  members. 

Data  and  Data  Use 

Three  years  of  PHI  data,  ending  in  May  2012,  have  been  loaded  to  the  CLG  proof-of-concept  data  mart.  The  team 
has  evaluated  the  data  and  is  developing  clinical  scenarios  meaningful  to  MHS  providers  for  training  and 
demonstration  purposes. 

A  local  EHIT  environment  is  maintained  for  testing  and  is  populated  with  test  data  mapped  to  the  MHS  data  model 
used  in  the  proof-of-concept. 

Air  Force  Population  Health  has  determined  that  the  data  it  has  supplied  for  this  project  is  covered  under  its  existing 
data  use  agreement. 

Dr.  Burke  at  WRNMMC  is  working  with  the  Walter  Reed  Institutional  Review  Board  (IRB)  in  collaboration  with 
the  Principle  Investigator  to  develop  policies  similar  to  those  in  effect  at  Montefiore  Medical  Center  governing  the 
use  of  Beneficiary  Encrypted  File  (BEF)  and  PHI  data  extracted  using  CLG  for  research,  quality  improvement, 
training,  and  direct  patient  care  at  Walter  Reed. 

Security 

The  CLG  team  has  reviewed  the  security  scan  report  provided  by  SPAWAR  and  developed  requirements  for 
necessary  application  changes.  These  changes  will  be  scheduled  for  implementation  over  the  next  two  to  three 
months  and  released  in  CLG  version  4. 3. See  objectives  1,  2,  and  5. 
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Training 

The  Clinical  Looking  Glass  end  user  training  materials  have  been  completely  revised  to  include  new  functionality 
developed  during  the  course  of  the  award  and  to  remove  functionality  sundowned  with  the  release  of  CLG  4. 1 .  The 
CLG  training  team  is  currently  creating  customized  end  user  documentation  for  the  MHS  CLG  proof-of-concept, 
using  MHS-relevant  clinical  examples  based  on  data  available  in  the  proof-of-concept  data  mart. 

Objective  1:  Data  Encryption 

The  CLG  environment  and  application  were  scanned  by  the  SPA  WAR  security  team  and  high  priority  items 
identified  for  remediation.  Environmental  settings  were  changed  in  compliance  with  high  priority  environmental 
items  and  the  and  application  tested.  Exceptions  will  be  requested  and  are  expected  to  be  granted  for  high  priority 
environmental  items  that  are  incompatible  with  CLG.  High  level  requirements  for  implementation  of  FIPS  140-2 
standard  for  data,  key  generation  and  management,  and  logging  were  developed,  analyzed,  and  estimated.  The 
development  work  necessary  to  support  these  requirements  will  be  scheduled  in  the  first  quarter  of  2013. 

Objective  Completion:  40% 

Objective  2:  Process  Improvement 

Threat  modeling  and  security  planning  have  been  incorporated  into  the  CLG  quality  assurance  lifecycle.  A  test  suite 
has  been  incorporated  into  the  required  regression  test  plan  to  exercise  known  risks  including  browser  hacks,  PHI 
exposure,  and  application  account  security.  Requirements  to  close  security  gaps  have  been  defined  and  the  highest 
priority  items  related  to  PHI  exposure  have  been  implemented. 

Objective  Completion:  60% 

Objective  3:  Data  Analysis  and  Upload  for  Review 

The  CLG  data  mart  housed  in  the  Air  Force  enclave  at  the  SPA  WAR  data  center  was  populated  with  MHS  encrypted 
data  and  mapped  to  the  application. 

Objective  Completion:  100% 

Objective  4:  DIACAP  Process  Management 

Since  development  of  the  original  scope  for  this  award,  it  has  been  determined  that  the  CLG  proof-of-concept  is  not 
subject  to  DIACAP  because  it  is  housed  inside  the  Air  Force  secure  enclave.  If  CLG  is  adopted  as  an  operational 
product  by  the  MHS  DIACAP  may  need  to  be  performed  at  that  time. 

Objective  Completion:  N/A 

Objective  5:  Usability  and  Security  Enhancements 

Requirements  gathering  for  more  granular  user  entitlements  and  restrictions  is  dependent  upon  user  acceptance 
testing  and  experimental  usage  of  CLG  as  currently  configured;  MHS  users  must  understand  CLG  usage  in  order  to 
specify  these  changes.  Due  to  the  data-loading  schedule,  training  of  the  first  classes  of  CLG  proof-of-concept  users 
has  not  yet  occurred.  This  training  is  anticipated  to  be  scheduled  during  the  first  quarter  of  2013. 

Objective  Completion:  0% 

Objective  6:  Component  Modernization 

R-Evolution  has  been  fully  integrated  into  CLG. 

Objective  Completion:  100% 
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Feature  Development 
Study  Designer 

Study  Designer  allows  users  to  create  studies  that  include  Time  to  Outcome,  List,  and  Time  in  Range  methods  and 
use  both  internally  specified  cohorts  and  uploaded  cohorts.  Development  work  to  further  enhance  Study  Designer 
functionality  occurred  during  this  reporting  period.  Study  Designer  was  included  in  the  CLG  4.1  update  to  the  MHS 
CLG  proof-of-concept  and  will  be  included  in  training  of  MHS  users. 

Scalability  Research 

Performance  and  scalability  requirements  were  defined  and  prioritized  based  on  clinical  scenarios.  The  CLG  team 
then  performed  technical  decomposition  to  identify  areas  for  research,  upgrade,  and  modification. 

The  highest  priority  performance  requirement  is  the  ability  to  append  many  outcomes  to  two  or  more  large  patient 
lists.  The  target  parameters  were  two  groups  of  100K  patients  each  with  fifty  outcomes.  CLG  4.1,  currently  in 
production  at  Montefiore  and  the  DoD  proof-of-concept,  supports  groups  of  this  size,  but  can  reliably  provide  results 
one  outcome.  To  accomplish  the  successful  run  of  up  to  fifty  outcomes,  changes  have  to  be  made  to  the  mart 
database,  the  processing  engine,  and  the  user  interface.  Some  of  these  changes  also  address  other  performance  goals, 
such  as  better  support  of  multiple  concurrent  users. 

Performance  and  Scalability  Work 

This  section  provides  details  of  the  performance  improvement  and  scalability  work  identified.  Completed  items  were 
done  by  or  before  the  sprint  ending  on  October  31,  2012.  All  effort  in  the  following  sprint  is  related  to  hardening  the 
application  for  release.  No  new  development  is  included.  Items  that  were  not  completed  before  October  3 1  are  to  be 
scheduled  within  the  next  six  months. 

24916  List  Complexity  (high-level  requirement,  parent  to  all  the  rest) 

Run  a  list  method  with  fifty  outcomes  on  two  patient  groups  of  100K  each. 

25506  Execute  multiple  analysis  definitions  in  parallel 

Take  advantage  of  parallel  processing  infrastructure  implemented  in  other  work  items  to  process  all  outcomes  at 
once  instead  of  sequentially. 

25507  Save  study  configuration  with  multiple  analysis  definitions  (complete) 

When  the  user  configures  a  list  method  with  up  to  fifty  outcomes,  the  system  must  save  that  configuration.  This 
work  item  parallelizes  and  improves  the  efficiency  of  some  back-end  tasks  so  that  studies  with  many  outcomes 
properly  save. 

25819  Collapsed/Expanded  state  of  ADs  in  List  Configurator  (complete) 

User  interface  enhancements  to  improve  the  management  of  many  outcomes. 

25523  Measure  performance  with  Oracle  llg  (complete) 

Compare  performance  of  key  application  queries  on  the  Oracle  llg  platform  to  the  performance  on  these  same 
queries  on  SQL  Server  2012.  Recommend  database  solution. 

25526  Modify  post  processing  engine  queries 

Redesign  the  queries  to  find  multiple  instances  of  patient  events  (e.g.,  all  lab  tests  or  admissions  during  a  given 
period)  and  that  store  the  cohort  data  to  take  better  advantage  of  SQL  Server  2012  features  and  therefore  improve 
processing  performance. 

25533  Move  cohort  data  to  module  build  results  table 

Addresses  technical  debt  incurred  during  development  of  event  collection  capabilities.  Unifies  the  storage  of  cohort 
and  event  collection  data  and  includes  regular  cleanup  of  aged  objects.  Improves  performance  of  multiple  users 
building  cohorts  concurrently. 
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25534  Implement  WCF  Broker  in  an  HPC  cluster 

Like  load  balancing,  the  WCF  Broker  assigns  tasks  across  multiple  nodes  for  parallel  processing.  This  improves 
performance  of  list  method  with  multiple  outcomes. 

25536  Break  workflow  into  small  activities  to  improve  cancellation  response  time 

When  the  user  cancels  running  a  study,  the  system  has  to  wait  until  the  current  task  completes  to  terminate  the  run. 
Breaking  the  workflow  into  more  smaller  tasks  will  create  more  possible  break  points  where  cancellation  can  take 
effect,  returning  resources  to  the  available  pool  faster. 

25537  Redesign  data  service  for  List/wide  view  (complete) 

Create  multiple  session  transformers  to  support  parallel  processing  in  data  services.  Combine  the  results  when  all  are 
complete.  Because  the  shape  of  the  resulting  data  is  different  for  wide  view  vs  long  view,  this  work  item  only 
addresses  wide  view  (the  default). 

25539  Migrate  TTO  to  New  data  services  workflow 

Refactor  TTO  patient  list  results  to  take  advantage  of  the  refactored  data  services  workflow.  Until  this  is  done,  the 
system  must  support  both  the  old  and  the  new  versions  of  data  services. 

25540  Migrate  TIR  to  new  data  services  workflow 

Refactor  TIR  patient  list  results  to  take  advantage  of  the  refactored  data  services  workflow.  Until  this  is  done,  the 
system  must  support  both  the  old  and  the  new  versions  of  data  services. 

25542  Evaluate  Silverlight  5  upgrade  (complete) 

Silverlight  5  will  better  support  the  configuration  of  up  to  fifty  outcomes  in  the  user  interface.  This  work  item 
included  full  regression  testing  with  Silverlight  5. 

25543  Remove  validation  of  analysis  definitions  from  study  designer  management  pane  loading 

When  the  user  accesses  Study  Designer,  the  application  validates  all  analysis  definitions  to  determine  their  fitness 
for  use  with  the  various  method  types.  Depending  on  how  many  analysis  definitions  the  user  has,  this  validation  can 
take  several  seconds.  It  is  not  necessary  until  the  user  is  ready  to  use  an  analysis  definition  in  a  method  (which,  in 
fact,  the  user  may  never  do).  This  work  item  will  move  the  validation  to  a  more  appropriate  point  in  the  workflow. 

25558  Install  and  measure  performance  of  SQL  Server  2012  (complete) 

Once  SQL  Server  was  identified  as  the  optimal  next-step  database  platform,  it  was  installed  and  fully  regression 
tested. 

26387  Export  large  data  sets  (upgrade  ASPose) 

ASPose  is  COTS  component  that  requires  upgrade  in  order  to  support  export  to  CSV  or  spreadsheet  of  groups  of 
more  than  300K  patients  with  more  than  twenty  outcomes.  CLG’s  output  generator  requires  some  modification  to 
work  with  the  upgraded  ASPose.  This  upgrade  will  also  allow  upgrade  of  data  services  and  the  output  generator  to 
64bit  -  these  components  are  currently  32bit  only  because  the  earlier  version  of  ASPose  requires  it. 

25541  Redesign  data  services  for  pivot 

Refactor  Pivot  to  take  advantage  of  the  refactored  data  services  workflow.  Until  this  is  done,  the  system  must 
support  both  the  old  and  the  new  versions  of  data  services. 

25538  Redesign  data  services  for  List  long  view 

Refactor  long  view  to  take  advantage  of  the  refactored  data  services  workflow.  Until  this  is  done,  the  system  must 
support  both  the  old  and  the  new  versions  of  data  services. 

25525  Manage  study  results  (Grid  with  sample) 

To  reduce  results  processing  time,  instead  of  displaying  all  list  results  on  screen,  display  a  sample.  Make  all  possible 
export  formats  immediately  available  so  that  the  user  need  not  wait  for  additional  processing  after  seeing  the  sample 
set. 
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Key  Research  Accomplishments 

•  Completion  of  load  of  PHI  data  on  SPAWAR  servers 

•  Study  Designer  release  for  use  by  all  users 

•  Position  of  public  trust  authorization  received  for  all  team  members  accessing  MHS  CLG  proof-of-concept 

Reportable  Outcomes 

•  Received  additional  JPC  funding  to  support  further  effort  through  2012. 

•  Applied  for  no-funds  extension  to  continue  project  through  2013. 

Conclusion 


At  the  close  of  this  year  we  are  on  the  verge  of  training  our  first  MHS  users  at  WRNMMC.  To  reach  this  goal  over 
the  course  of  this  year  we  have  relied  upon  our  relationship  with  Air  Force  Population  Health  and  the  SPAWAR  data 
center  to  support  CLG  and  provide  MHS  data  to  its  data  mart.  During  the  reporting  period  the  CLG  servers  were 
moved  out  of  the  Washington  Navy  Yard  data  center  along  with  the  rest  of  the  Air  Force  enclave.  This  move  was  not 
significantly  disruptive  to  our  progress.  Population  Health’s  commitment  to  the  project  was  exemplified  by  the 
subsequent  load  of  expanded  PHI  data  to  the  CLG  data  mart. 

The  team  underwent  some  churn  during  the  reporting  period.  Several  software  engineers  departed  and  were  replaced. 
The  Senior  Clinical  Analyst,  responsible  for  training  and  end  user  mentoring,  departed  and  a  replacement  identified. 
Several  months  of  development  were  allocated  to  performance  and  scalability  improvements.  Release  4.1  of  CLG 
was  rolled  out  to  the  Montefiore  community  and  subsequently  installed  on  the  proof-of-concept  and  fully  tested 
there. 

The  CLG  PHI  proof-of-concept  is  poised  to  be  revealed  to  MHS  users.  The  CLG  team  eagerly  awaits  the 
opportunity  to  train  members  of  this  community  in  the  use  of  this  powerful  tool. 

References 

None  Cited. 

Appendix 

List  of  CLG  team  members  whose  salaries  were  wholly  or  partially  funded  by  this  contract.  Percentages  of  salary 
vary  due  to  shifting  participation  and  effort  as  personnel  left  and  joined  the  team. 


CSI 

Chowdhury,  Soma 

0-40% 

Edwards,  Richie 

25-30% 

Geberer,  Noah 

60-70% 

Golden,  Joe 

15-30% 

Lee,  William 

0-15% 

McCroskey,  Mia 

29-50% 

Muresan,  Mircea 

0-30% 

Patel,  Saurabh 

10-30% 
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Paul,  Lisha 

15-30% 

Powers,  Ken 

20-60% 

Srivastava,  Navneet 

20-30% 

Swamy,  Harish 

10-20% 

Zappulla,  Ron 

45-50% 

Luis  Lira 

100% 

Weissman,  Judith 

0-80% 

Ding,  Jinlin 

20% 

Mukerji,  Himadri 

10-100% 

James  Lee 

30-40% 

JPC 

Castro,  Moises 

0-100% 

Abramov,  Margarita 

100% 

Yousef,  Jason 

100% 
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